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ABSTRACT 



This paper discusses the evaluation of electronic resources. 
The first section addresses the importance of a clear policy framework, 
including the goals of a survey on higher education research and policy 
issues concerning online education, instructional technology, and distance 
learning. The second section discusses the importance of a robust methods 
framework. This section describes the following four strands of a three-year 
project in the United Kingdom with the purpose of developing a national user 
behavior monitoring and evaluation framework: a general survey of end users 
of all electronic information services; a real time survey of .the use of JISC 
(Joint Information Systems Committee) -funded services; a general survey of 
provision; and a qualitative longitudinal monitoring of use. Key issues 
highlighted by the project are summarized. (Contains 22 references.) (MES) 
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Introduction 

Evaluating the impact of access to electronic information is a key challenge globally for memory 
institutions such as archive, libraries, and museums. This is because of their need to carefully 
manage limited resources to best serve users' needs, and to justify to purse-holders that this 
investment represents good value for money. As these purse-holders are often educational 
institutions or government departments, value for money must generally be measured not in value 
to an individual user but in terms of contribution to the creation of a knowledge economy. 

Too often evaluation of access to electronic resources is bogged down in the processes of 
managing that access. For example, an interim evaluation of the UK's National Electronic Site 
Licensing Initiative (Richardson et al. 2000) produced the following major conclusions: 

• NESLI has achieved much in a rapidly-changing and complex environment. It has done 
very well in difficult circumstances. 

• There is widespread support from all stakeholders for the principle of a single negotiating 
body for e-joumals on behalf of all UK universities. NESLI constitutes a significant 
advance in the way new business deals are transacted in the developing electronic 
marketplace and is being closely monitored in many other countries. 

• Progress in agreeing deals, getting them accepted by libraries and achieving end user take 
up has been slow because of the complexity of the process. Nevertheless there are signs 
of gathering momentum and support from all the necessary stakeholders. 
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• Although it needs to be negotiated in each case and the legal complexities can be 
daunting, the model licence has been a major success in providing the framework for 
significant deals. 

• The offers that have been made have been of value to some libraries but the different 
situations of the libraries means that a single offer cannot meet all needs. 

• Evaluating complex deals in a limited timescale has led to many libraries being unable to 
take up offers. 

• Offering the service through a single portal has advantages in providing one access point 
for end users but is seen as a restrictive practice by many stakeholders concerned about a 
single service undertaking all activities. 

The point here is not about the NESLI service — it is that negotiating and managing access to 
electronic resources is such a complicated business that our evaluation activities rarely penetrate 
to actual user experiences. Sometimes we get hints of the widespread managerial changes that 
are revolutionising information professions, and there is increasing anecdotal evidence to suggest 
that some users are by-passing traditional printed resources if they possibly can. What we need to 
evaluate is whether people learn from, and use, information in a new way when they can get at it 
electronically. 

Two key things are needed to help us evaluate access to electronic resources more effectively. 

The first is a clear policy framework. The second is a robust methods framework. 

A Clear Policy Framework 

Every institution, every region, every country will have a different set of policy pressures that 
might be applied to the evaluation of access to electronic resources. For example, in the UK the 
government is investing in Further Education colleges to ensure that each has Internet 
connections and access to high-quality electronic information and materials. Key objectives for 
the Further Education sector as a whole include increasing participation from communities that 
are traditionally socially-excluded, increasing retention rates for students who begin courses, and 
improving rates for their successful conclusion of these courses. 

It would be easy for these pre-existing policy pressures to be applied simply to the evaluation of 
electomic resources. For example, in the UK we might be asked what percentage increase in 
retention rates will be achieved by spending £100,000 pounds on access to electronic learning 
materials. 

The answers would not necessarily get any of us very far. The answers would depend on the very 
mass of management issues they we should be trying to penetrate: where would the money come 
from, how would awareness of the available resources be promoted, how would information 
professionals and course lecturers be trained to feel confident in their use, etc., etc., etc. 

The challenge is on: information professionals must be part of a wider policy debate about the 
ways in which electronic accesss to information can/should be expected to contribute to the 
creation of a knowledge economy. For example, routine access to electronic resources for all 
would facilitate the learning and sharing of basic computing skills. 

There are many efforts underway to stimulate these policy debates, but it is unclear how the 
varied communities of practice with a shared interest are joining their efforts together. Are policy 
advisors taking a lead? Are educational or economic think tanks? Are library associations? 
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One interesting experiment that is underway is spear-headed by the Andrew W. Mellon 
Foundation in the US. It is conducting a survey on higher education research and policy issues 
concerning online education, instructional technology, and distance learning. The outcomes of 
this exercise should be enlightening, and could be usefully modelled in other countries. Key 
goals of the survey are to guage: 

1 . The greatest issue of concern regarding the use of instructional technology in higher 
education. 



Whether the use cf instructional technology in on-campus contexts if believed to affect 



the quality of teaching. 



3. The perceived promise of distance education. 

4. Whether instructional technology may enhance access to or equity in higher education, or 
has the potential to grow enrollment and reaching/marginalising new types of students. 

5. The characteristics of high-quality and effective online learning. 

6. Perceived potential impacts that the use of instructional technology might have on 
academic values, processes, or governance. 

7. The economic benefits, if any, to using instructional technology, and the opportunity 
costs of large investments in instructional technology and distance leaeming. 



8. The potential impacts on existing campus resources. 

9. The likely changes that the use of instructional technology are likely to bring, and likely 
sources of resistance to these changes. 

10. Valuable avenues for empirical research on the use of instructional technology, the 
growth of online education, and the emergence of the new distance learning ventures. 



A Robust Methods Framework 

Once the challenges to be addressed with access to electronic resources are clear, then the right 
questions can be asked. It is then that a robust methods framework will be essential for 
measuring progress toward our goals. 

Explicit evaluation methodology has been the focus of many projects looking at institutions (e.g. 
Lubans 2000, Parnell 2001, Squires 2001), media types (e.g. Ehrmann 1997), or specific subject 
areas (e.g. Porter and Greenstein 1997). 

Delving deeper into the literature of any discipline is likely to reveal a range of articles on 
computer applications and their impact on scholarship. Evaluating access to electronic resources 
will clearly need to involve reflective practitioners such as these. For example, in my own 
discipline of archaeology the literature has seen first the introduction of computing applications, 
then excited singing from the choir of the convinced, followed by a rather pessimistic backlash, 
and finally by more mature reflection and evaluation (Aldenderfer 1998, Allen et al. 1990, 

Bewley et al. 1999, Clark 1996, Gillings and Goodrick 1996, Gillings and Wise 1999, Lock 1995, 
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Miller and Richards 1995, Orton 1999, Richards and Robinson 2000, Wheatley 1993, and Wise 

2001 ). 



In the UK a three-year project has been funded with the explicit purpose of developing a national 

User Behaviour Monitoring and Evaluation Framework (Rowley 2000). 

The project has four strands, all due to be complete in July 2002. These are: 

A. General survey of end users of all electronic information services described as a single broad- 
based sample survey of patterns of use and non-use of all electronic information services by 
librarians, academics and students administered through face-to-face or telephone interviews 
to ensure accurate sampling of all relevant sub-populations. 

B. A ' real time ’ survey of the use of JlSC-funded services through a combination of Web based 
questionnaires, session logs, and user password data to profile users. 

C. General survey of provision thorugh a combination of a Web survey of resource access 
provided by individual universities, with telephone polls of purchasing intentions, backed up 
by a small number of face to face interviews with key informants to profile service provision. 

D. A qualitative longitudinal monitoring of use through an ongoing programme of longitudinal 

qualitative evaluation centred around a selection of institutions and subject, and an associated 
programme of one-off studies on the behaviour and needs of specific disciplinary groups. 

Preliminary results (Rowley 2000) from the first year of this monitoring and evaluation 

framework are promising, although they inevitably highlight the many complexities and 

managerial issues still facing information professionals. 

Key issues highlighted include: 

1. The diversity of the UK university community. There are institutions where staff and 
students have very limited access, and library web sites are only just being established. 
There are academic staff who demonstrate innovative approaches to the integration of 
electronic resources in learning, and there are other staff who believe that electronic 
resources have no relevance for their activities. National initiatives, specifically 
provision of the JANET network and data centres hosting commercially-licenced 
resources, are cited as clearly having some impact in levelling the national playing field. 

2. Key policy issues include government agenda on widening access and participation in 
education and regular, rigourous quality reviews of educational provision. These 
influence attitudes about access to electronic resources and can present problems. 

3. Managerial issues facing library and information services are very serious. Collection 
development is frustrated by changing access and purchase arrangements for sources 
such as electronic journals. Budgets devolved from the library to departments further 
complicates matters. 

4. Academic staff act as key 'gatekeepers’ to student learning, and this privileged 
relationship should be drawn upon. It is essential to build their skills and confidence. 

5. General Web search engines and known Web sites are the first resort for most academic 
queries, as well as for many personal searches, done by undergraduate students. There is 
little if any evidence to suggest that they use the many tools designed by their institution 
library and information service to facilitate their general searching, or that they appreciate 
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the potential time savings offerred by these tools. General Web skills tuition needs to 
continue or be increased. 

6. Graduate students often have a well developed knowledge of specific sources relevant to 
their studies, but in other respects do not appear to use electronic resources in a 
substantially different way than undergraduate students. 

7. Use of commercially-licensed databases and journals was low. 

One of the deliverables from this interesting project will be a transferable 'toolkit' for evaluating 
the stage of an institution's development in terms of facilitating access to, and use of, electronic 
resources. 

This monitoring and evaluation framework project has already had the welcome affect of 
stimulating some debate about evaluation and the role of academic resources in the UK higher 
education community (e.g. Lonsdale and Urquart 2001) and will no doubt continue to stimulate 
and challenge in upcoming years. Greater international co-ordination of similar initiatives would 
be very welcome, as would debate in the library and information community. 

Another key area of development is in obtaining fair, comparable usage statistics about online 
resources. Much of this work is stimulated by library requirements to justify/challenge high 
subscription prices for electronic subscriptions. This area is a minefield, however, with widely 
varying practices used by publishers and libraries to record, measure, and report usage statistics. 
Some high-profile groups have taken the daring step of publishing conclusions about user 
behaviour based on usage statistics currently available. While these conclusions may be borne 
out in time, it seems sensible to await further implementation of the ICOLC Usage Statistics 
Guidelines (ICOLC 1998) and detailed implementation studies being undertaken by the PALS 
group in the UK (PALS Usage Statistics Working Group 2001), and the ARL e-metrics project in 
the US. 
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